Methods and systems for multiple access to a single hardware data stream

ABSTRACT

A target is outputted to an ideal position in 3D space. A viewer indicates the apparent position of the target, and the indication is sensed. An offset between the ideal and apparent positions is determined, and an adjustment determined from the offset such that the apparent position of the ideal position with the adjustment matches the ideal position without the adjustment. The adjustment is made to the first entity and/or a second entity, such that the entities appear to the viewer in the ideal position. The indication may be monocular with a separate indication for each eye, or binocular with a single viewer indication for both eyes. The indication also may serve as communication, such as a PIN input, so that calibration is transparent to the viewer. The method may be continuous, intermittent, or otherwise ongoing over time.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.14/874,313, filed Oct. 2, 2015, which is hereby incorporated byreference for all purposes.

FIELD OF THE INVENTION

This invention relates to output of three dimensional images. Moreparticularly, this invention relates to approaches for individuallycalibrating a three dimensional display system such as a stereo displaysystem to accommodate a viewer's particular visual parameters.

DESCRIPTION OF RELATED ART

Humans perceive depth in part through stereo vision, noting thedifference in perspective between the left and right eyes when viewingthe physical world. Stereo displays take advantage of this feature: bydisplaying different images to the left and right eyes, depth can beportrayed even if both images are two-dimensional.

However, human vision may not be uniform from one individual to another,and/or for an individual over time. For example, differences in geometrysuch as the spacing between a person's eyes, the relative positions ofthe eyes, the physical structures of the eyes, the alignment of the eyeswith one another, etc. can affect the way a scene appears to anindividual. Thus, individuals viewing a physical scene may perceive thatscene differently, based on the particulars of each individual's eyes.Even if two people both perceive an object to be at a distance of (forexample) 50 cm, What those two people see in making that determinationmay vary.

Because of this, a “one size fits all” stereo display configuration maybe problematic. Displayed imagery that does not correspond with what aviewer sees when looking at a physical environment may consciously orunconsciously be perceived as unrealistic or unconvincing. In addition,variations in how a stereo display scene appears compared with how thephysical world appears from the viewer's perspective, errors (or atleast apparent errors) in how the scene is displayed may contribute(again, with or without the viewer's awareness) to issues such asdisorientation, eyestrain, nausea, etc.

BRIEF SUMMARY OF THE INVENTION

The present invention contemplates a variety of systems, apparatus,methods, and paradigms for calibrating three dimensional display systemsto individual viewers.

In one embodiment of the present invention, a machine-implemented methodis provided that includes, in a processor, outputting a first entity inan ideal position in three-dimensional space, and sensing a viewerindication of an apparent position of the first entity. The method alsoincludes determining an offset between the ideal position and theapparent position, and determining an adjustment from the offset, suchthat an apparent position of the ideal position with the adjustment issubstantially similar to the ideal position absent the adjustment.Thereafter the adjustment is applied the first entity and/or a secondentity.

Outputting the first entity in the ideal position may include outputtingthe first entity to first and second displays of a stereo display pair.Outputting the first entity in the ideal position may include outputtingthe first entity in a first ideal position to a first display of astereo display pair, and outputting the first entity in a second idealposition to a second display of the stereo display pair. The first andsecond ideal positions may be substantially two-dimensional.

Outputting the first entity in the ideal position may include outputtingthe first entity in a first ideal position to a first eye of the viewerbut not a second eye of the viewer, and outputting the first entity in asecond ideal position to the second eye of the viewer but not the firsteye of the viewer.

The method may include outputting the first entity substantiallysimultaneously to the first eye and the second eye, wherein sensing theviewer indication of the apparent position of the first entity comprisessensing a substantially simultaneous viewer first indication of a firstapparent position of the first entity to the first eye and viewer secondindication of a second apparent position of the first entity to thesecond eye.

The method may include outputting the first entity sequentially to thefirst eye and the second eye, wherein sensing the viewer indication ofthe apparent position of the first entity comprises sequentially sensinga viewer first indication of a first apparent position of the firstentity to the first eye, and a viewer second indication of a secondapparent position of the first entity to the second eye.

The method may include outputting substantially nothing to the secondeye while outputting the first entity to the first eye, and outputtingsubstantially nothing to the first eye while outputting the first entityto the second eye. The method may include substantially obstructing thesecond eye while outputting the first entity to the first eye, andsubstantially obstructing the first eye while outputting the firstentity to the second eye.

The viewer indication may include a viewer posture and/or a viewergesture. The viewer indication may include the viewer substantiallyaligning an end-effector with the apparent position of the first entity.

The viewer first indication may include the viewer substantiallyaligning a first end-effector with the first apparent position of thefirst entity, and the viewer second indication may include the viewersubstantially aligning the first end-effector with the second apparentposition of the first entity, substantially simultaneously to the viewersubstantially aligning the first end-effector with the first apparentposition of the first entity.

The viewer first indication may include the viewer substantiallyaligning a first end-effector with the first apparent position of thefirst entity, and the viewer second indication may include the viewersubstantially aligning the first end-effector with the second apparentposition of the first entity.

The viewer first indication may include the viewer substantiallyaligning a first end-effector with the first apparent position of thefirst entity, and the viewer second indication may include the viewersubstantially aligning a second end-effector with the second apparentposition of the first entity.

The entity may include a virtual reality entity and/or an augmentedreality entity.

The method may include outputting the first entity to a stereo display.The method may include outputting the first entity to a see-throughdisplay.

The method may include sensing the viewer indication of the apparentposition through stereo imaging. The method may include sensing theviewer indication of the apparent position through depth imaging.

The viewer indication may include a viewer communication with theprocessor in addition to the indication of the apparent position. Theviewer communication may include: activating a device in communicationwith the processor, activating a data entity comprising executableinstructions instantiated on the processor, waking the device incommunication with the processor, waking the data entity comprisingexecutable instructions instantiated on the processor, unlocking thedevice in communication with the processor, unlocking the data entitycomprising executable instructions instantiated on the processor,addressing the device in communication with the processor, addressingthe data entity comprising executable instructions instantiated on theprocessor, identifying a user of the device in communication with theprocessor, identifying the user of the data entity comprising executableinstructions instantiated on the processor, entering a securityverification for the device in communication with the processor, and/orentering the security verification for the data entity comprisingexecutable instructions instantiated on the processor.

In another embodiment of the present invention, a machine-implementedmethod is provided that includes, in a processor, establishing asubstantially three-dimensional combined ideal position, determiningfrom the combined ideal position a substantially two-dimensional firstideal position for a first display of a stereo display pair, anddetermining from the combined ideal position a substantiallytwo-dimensional second ideal position for a second display of thedisplay stereo pair. The method also includes outputting a first entitysubstantially to the first ideal position in the first display, andoutputting the first entity substantially to the second ideal positionin the second display. The method further includes sensing a viewerindication of an apparent position, determining a first position offsetsubstantially representing a difference between the apparent positionand the first ideal position, determining a second position offsetsubstantially representing a difference between the apparent positionand the second ideal position, and determining an adjustment for thecombined ideal position from the first and second position offsets, suchthat an apparent position of the combined ideal position with theadjustment is substantially similar to the ideal position absent theadjustment. The method includes applying the adjustment to a threedimensional position of the first entity and/or a second entity, andoutputting the first entity and/or second entity with the adjustmentapplied thereto to the stereo display pair.

The method may include sequentially outputting the first and secondentities, outputting the first entity in the first display while notoutputting the second target in the second display, and outputting thesecond entity in the second display while not outputting the first,target in the first display. The method may include outputting the firsttarget in the first display while outputting substantially nothing inthe second display, and outputting the second target in the seconddisplay while outputting substantially nothing in the first display. Themethod may include outputting the first target in the first displaywhile substantially obstructing the second display, and outputting thesecond target in the second display while substantially obstructing thefirst display.

Sensing the viewer indication may include sensing a viewer firstindication of the first target and a viewer second indication of thesecond target.

The method may include outputting the first target in the first displayand substantially simultaneously outputting the second target in thesecond display.

Sensing the viewer indication may include sensing substantiallysimultaneous indication of the first and second targets.

The viewer indication may include a viewer posture and/or a viewergesture. The viewer indication may include the viewer substantiallyaligning an end-effector with the apparent position of the first entity.

The viewer first indication may include the viewer substantiallyaligning a first end-effector with the first apparent position of thefirst entity, and the viewer second indication may include the viewersubstantially aligning the first end-effector with the second apparentposition of the first entity, substantially simultaneously to the viewersubstantially aligning the first end-effector with the first apparentposition of the first entity.

The viewer first indication may include the viewer substantiallyaligning a first end-effector with the first apparent position of thefirst entity, and the viewer second indication may include the viewersubstantially aligning the first end-effector with the second apparentposition of the first entity.

The viewer first indication may include the viewer substantiallyaligning a first end-effector with the first apparent position of thefirst entity, and the viewer second indication may include the viewersubstantially aligning a second end-effector with the second apparentposition of the first entity.

The viewer first indication may include the viewer substantiallyaligning a first end-effector with the first apparent position of thefirst entity, and the viewer second indication may include the viewersubstantially aligning the first end-effector with the second apparentposition of the first entity, substantially simultaneously to the viewersubstantially aligning the first end-effector with the first apparentposition of the first entity.

The entity may include a virtual reality entity and/or an augmentedreality entity.

The first and second displays are see-through displays.

The method may include sensing the viewer indication of the apparentposition through stereo imaging. The method may include sensing theviewer indication of the apparent position through depth imaging.

In another embodiment of the present invention, an apparatus is providedthat includes a processor, a 3D display in communication with theprocessor, the display being adapted to output a first entity at anideal position, and a sensor in communication with the processor, thesensor being adapted to determine a viewer indication of an apparentposition of the first entity. An offset determiner is instantiated onthe processor, the offset determiner including executable instructions,the difference determiner being adapted to determine an offset betweenthe ideal position and the apparent position. An adjustment determineris instantiated on the processor, the adjustment determiner includingexecutable instructions, the adjustment determiner being adapted todetermine an adjustment from the offset such that an apparent positionof the ideal position with the adjustment is substantially similar tothe ideal position absent the adjustment. An adjustment applier isinstantiated on the processor, the adjustment applier includingexecutable instructions, the adjustment applier being adapted to applythe adjustment to at least one of a group consisting of the first entityand a second entity.

The display may be a stereo display. The display may be a see-throughdisplay. The sensor may include an imager. The sensor may include astereo imager pair. The sensor may include a depth sensor and/or depthimager. The processor, the display, and the sensor may be disposed on ahead-mounted display.

In another embodiment of the present invention, an apparatus is providedthat includes a processor, a stereo display pair in communication withthe processor, the stereo display pair comprising first and seconddisplays, the stereo display pair being adapted to output at least oneentity, and a sensor in communication with the processor, the sensorbeing adapted to determine a viewer indication of an apparent position.A position establisher is instantiated on the processor, the positionestablisher including executable instructions, the position establisherbeing adapted to establish a substantially three-dimensional combinedideal position. A stereo position determiner instantiated on theprocessor, the stereo position determiner including executableinstructions, the stereo position determiner being adapted to determinea substantially two-dimensional first ideal position in the firstdisplay substantially corresponding with the combined ideal position,and to determine a substantially two-dimensional second ideal positionfor in the second display substantially corresponding with the combinedideal position. An outputter is instantiated on the processor, theoutputter including executable instructions, the outputter being adaptedto output a first target substantially to the first ideal position inthe first display, and to output a second target substantially to thesecond ideal position in the second display. An offset determiner isinstantiated on the processor, the offset determiner includingexecutable instructions, the offset determiner being adapted todetermine a first position offset substantially representing adifference between the apparent position and the first ideal position,and to determine a second position offset substantially representing adifference between the apparent position and the second ideal position.An adjustment determiner is instantiated on the processor, theadjustment determiner including executable instructions, the adjustmentdeterminer being adapted to determine an adjustment for the combinedideal position from the first and second position offsets such that anapparent position of the ideal position with the adjustment issubstantially similar to the ideal position absent the adjustment. Anadjustment applier is instantiated on the processor, the adjustmentapplier including executable instructions, the adjustment applier beingadapted to apply the adjustment to at least one the entity.

The first and second displays may be see-through displays. The sensormay include an imager. The sensor may include a stereo imager pair. Thesensor may include a depth sensor and/or depth imager.

In another embodiment of the present invention, an apparatus is providedthat includes means for establishing a substantially three-dimensionalcombined ideal position, means for determining from the combined idealposition a substantially two-dimensional first ideal position for afirst display of a stereo display pair, and means for determining fromthe combined ideal position a substantially two-dimensional second idealposition for a second display of the display stereo pair. The apparatusincludes means for outputting a first target substantially to the firstideal position in the first display, means for outputting a secondtarget substantially to the second ideal position in the second display.The apparatus also includes means for sensing a viewer indication of anapparent position, means for determining a first position offsetsubstantially representing a difference between the apparent positionand the first ideal position, means for determining a second positionoffset substantially representing a difference between the apparentposition and the second ideal position, and means for determining anadjustment, such that an apparent position of the ideal position withthe adjustment is substantially similar to the ideal position absent theadjustment. The apparatus further includes means for applying theadjustment to a three dimensional position of at least one outputentity; and means for outputting the at output entity to the stereodisplay pair.

In another embodiment of the present invention, a method is providedthat includes outputting to a stereo display of a head mounted display acalibration target in an ideal position in three-dimensional space. Themethod includes sensing with a depth imager of the head mounted displaya viewer indication of the apparent position of the first entity,determining computationally in a processor of the head mounted displayan offset between the ideal position and the apparent position, anddetermining computationally in the processor an adjustment from theoffset such that the apparent position of the ideal position with theadjustment is substantially similar to the ideal position absent, theadjustment. The method also includes thereafter applying the adjustmentin the processor to the calibration target and/or a data entity, andoutputting the calibration target and/or data entity with the adjustmentapplied thereto to the stereo display.

In another embodiment of the present invention, a method is providedthat includes, in a processor of a head mounted display, computationallydetermining a substantially three-dimensional combined ideal positionfor a calibration target, and in the processor computationallydetermining from the combined ideal position a substantiallytwo-dimensional first ideal position for a first display of a stereodisplay pair of the head mounted display, and in the processorcomputationally determining from the combined ideal position asubstantially two-dimensional second ideal position for a second displayof the stereo display pair. The method includes outputting said targetsubstantially to the first ideal position in said first display,outputting the target substantially to the second ideal position in thesecond display, and sensing with a sensor of the head mounted display aviewer indication of an apparent position of the target. The method alsoincludes in the processor computationally determining a first positionoffset substantially representing a difference between the apparentposition and the first ideal position, in the processor computationallydetermining a second position offset substantially representing adifference between the apparent position and the second ideal position,and in the processor computationally determining an adjustment for thecombined ideal position from the first and second position offsets, suchthat an apparent position of said combined ideal position with saidadjustment is substantially similar to said ideal position absent saidadjustment. The method further includes in the processor applying theadjustment to a three dimensional position of the target, and outputtingthe calibration target with the adjustment applied thereto to the stereodisplay pair.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Like reference numbers generally indicate corresponding elements in thefigures.

FIG. 1 shows a top down schematic for an example three dimensionaldisplay system according to the present invention, with an exampleinter-pupillary distance and apparent position.

FIG. 2 shows a top down schematic for another example three dimensionaldisplay system according to the present invention, with a differentinter-pupillary distance and apparent position from FIG. 1.

FIG. 3 shows apparent positions of images for the left and right eyes ofa viewer, for an example inter-pupillary distance.

FIG. 4 shows apparent positions of images for the left and right eyes ofa viewer, for a different example inter-pupillary distance from FIG. 3.

FIG. 5 shows an example method for individualized three dimensionaldisplay calibration according to the present invention, in flow-chartform.

FIG. 6 through FIG. 13 show a sequence as might correspond to at leastcertain features of an example method for individualized threedimensional display calibration according to the present invention,illustrated as might be seen and/or carried out by a viewer.

FIG. 14 shows another example method for individualized threedimensional display calibration according to the present invention,explicitly binocular in nature, in flow-chart form.

FIG. 15 shows another example method for individualized threedimensional display calibration according to the present invention,explicitly monocular in nature, in flow-chart form.

FIG. 16 shows another example method for individualized threedimensional display calibration according to the present invention,explicitly establishing positions and outputting entities, in flow-chartform.

FIG. 17 shows another example method for individualized threedimensional display calibration according to the present invention, withexample references to approaches for execution of steps thereof, inflow-chart form.

FIG. 18 shows an example apparatus for individualized three dimensionaldisplay calibration according to the present invention, in schematicform.

FIG. 19 shows another example apparatus for individualized threedimensional display calibration according to the present invention, inschematic form.

FIG. 20 shows another example apparatus for individualized threedimensional display calibration according to the present invention, inperspective view.

FIG. 21 shows a block diagram of a processing system that may implementoperations of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

With reference to FIG. 1, a top down schematic for a three dimensionaldisplay system is shown therein. Left and right eyes 102 and 104 areshown, with left and right displays 106 and 108 of a three dimensionaldisplay system (as shown a stereo display, though this is an exampleonly) disposed in front of the left and right eyes 102 and 104respectively. Left and right images 110 and 112 are outputted to thedisplays 106 and 108. Sight lines 114 and 116 are shown extending fromthe eyes 102 and 104 to the images 110 and 112.

Because the images 110 and 112 are outputted to the displays 106 and108, sight lines 114 and 116 to the images 110 and 112 terminate at thedisplays 106 and 108. However, human stereo vision typically combinesthe two images 110 and 112, interpreting images 110 and 112 as a singletarget 122. The position of the target 122 is determined by extendingvirtual sight lines 118 and 120 from the sight lines 114 and 116 to apoint of convergence. Thus even though the images 110 and 112 themselvesmay be substantially or entirely two-dimensional, and/or constrained topositions in two-dimensional space (e.g. displayed on two-dimensionalsurfaces), entities appearing to a viewer as being three-dimensionaland/or as being in three-dimensional space (e.g. at some depth differentfrom the distance to the two-dimensional displays) nevertheless may bedisplayed.

Turning to FIG. 2, another top down schematic for a three dimensionaldisplay system is shown therein. As in FIG. 1, left and right eyes 202and 204 are shown in FIG. 2. However, as may be seen the eyes 202 and204 in FIG. 2 are farther apart from one another than the eyes 102 and104 in FIG. 1. The spacing between eyes may be referred to as theinter-pupillary distance or IPD, i.e. the distance between the centersof the pupils. Inter-pupillary distance may vary significantly betweenindividuals. For example, while a mean for young adult U.S. males isapproximately 65 millimeters, values as low as 52 mm or as high as 78 mmare not unknown, and greater extremes may exist. Inter-pupillarydistance also may vary with age (typically being smaller in childrenthan in adults), due to certain injuries, etc. It is noted that theinter-pupillary distance in FIG. 2 is about 15% greater than that inFIG. 1, while the maximum real-world value of 78 mm cited above is 50%greater than the minimum real-world value of 52 mm cited above. Thus thevariation between FIG. 1 and FIG. 2 is not necessarily an exaggerationbeyond anticipated human anatomical norms in that regard.

Continuing in FIG. 2, left and right displays 206 and 208 of a threedimensional display system are disposed in front of the left and righteyes 202 and 204, with left and right images 210 and 212 outputtedthereto. Sight lines 214 and 216 extend from the eyes 202 and 204 to theimages 210 and 212, and virtual sight lines 218 and 220 converge suchthat a viewer would see a target 222 at a location in three dimensionalspace.

As may be seen by comparison of FIG. 2 with FIG. 1, the distance fromthe viewer in FIG. 2 (as represented by eyes 202 and 202) to the target222 therein is visibly less than the distance from the viewer in FIG. 1to the target 122 therein. As may also be seen by comparison, the size,shape, configuration, and position of the displays 206 and 208 in FIG. 2are otherwise similar to those of the displays 106 and 108 in FIG. 1,and that the positions of the images 210 and 212 in FIG. 2 are similarto the positions of the images 110 and 112 in FIG. 1. Yet although thedisplay systems in FIG. 1 and FIG. 2 are at least extremely similar, theapparent position of the targets 122 and 222 are visibly quitedifferent; namely, the image 222 in FIG. 2 appears to be at a closerdistance than the image 122 in FIG. 1.

Then, when an entity is outputted by a three dimensional system such asa stereo display system, where that entity appears to a viewer may varyfrom one viewer to the next. In other words, what a viewer perceives asthe apparent position of the entity may be different from an ideal orintended position for the entity.

Such variation may occur even without three dimensional displays, duefor example to factors such as the aforementioned variation ininter-pupillary distance. For example, as may be seen in FIG. 3, aviewer with an inter-pupillary distance and in an arrangement similar towhat is shown in FIG. 1 may see image positions similar to 310 and 312shown in FIG. 3 with his or her left and right eyes. In FIG. 3, thelarger center mark represents a perpendicular away from the viewer (i.e.“straight ahead”) while the surrounding marks indicate intervals of 15degrees in various directions. Thus, the left image 310 appearsapproximately 9 degrees right of perpendicular viewed from the left eye,while the right image 312 appears approximately 9 degrees left ofperpendicular viewed from the right eye.

With reference to FIG. 4, a viewer with an inter-pupillary distance andin an arrangement similar to what is shown in FIG. 2 might see imagepositions similar to 410 and 412. The left image 410 appearsapproximately 22 degrees right of perpendicular viewed from the lefteye, while the right image 412 appears approximately 22 degrees left ofperpendicular viewed from the right eye.

In unassisted human vision, differences such as those visible betweenFIG. 3 and FIG. 4 may not be of concern; a real-world target in threedimensional space “is where it is”, and individuals may learn to judgedistances and positions for targets based on the particulars of theirown eyes and brain. The precise deflections (and/or the angles ofconvergence those deflections represent) seen by each eye may vary fromperson to person, but over time each person may learn (perhaps notconsciously) what deflections/angles correspond with what positions inthree dimensional space.

However, when entities are displayed without necessarily existingphysically, such as may be the case for at least some virtual realityand/or augmented reality content, such individual variations may proveproblematic. A display system may output images 110 and 112 as in FIG. 1with the expectation that, the viewer would perceive a target 122 at thedistance shown in FIG. 1, based on an idealized assumption that theinter-pupillary distance of the viewer also is as shown in FIG. 1.However, for a viewer with an inter-pupillary distance similar to thatin FIG. 2, the apparent position of the target 222 may be as shown inFIG. 2, and thus substantially closer than the ideal position of thetarget 122 in FIG. 1.

In such instance, to a viewer with an inter-pupillary distance as inFIG. 2 the target 222 may appear much closer than is intended. Morebroadly, variations in inter-pupillary distance and/or other factors maycontribute to a perception by viewers that objects displayed in (forexample) virtual reality and/or augmented reality systems are “in thewrong place”. Such discrepancies between ideal positions and apparentpositions may cause outputted entities to appear differently than isexpected, in addition to and/or instead of variations in perceiveddistance only. Scenes may appear distorted from how they are anticipatedto be displayed, for example. More colloquially, the output beingdisplayed may appear “wrong” (even though the viewer may not necessarilybe able to identify precisely what is wrong with the output).

Even if relatively small, errors in apparent distance may beproblematic. For example, interacting with a three dimensionalenvironment may be more difficult if objects are not at the positionsthat the viewer expects from the appearance of those objects. This maybe particularly significant for three dimensional environments that arerelatively rich and sophisticated, e.g. with many objects and/orphenomena displayed therein, objects and/or phenomena moving, affectingone another, etc. In addition, a high level of interactivity and/oropportunity to manipulate such environments, responsiveness to finecontrol, etc. may contribute to position errors (or at least perceivederrors) and so forth being particularly problematic. Furthermore, if anenvironment is to be realistic, immersive, etc. a discrepancy betweenwhere objects “should be” for a given viewer and where they are as faras the three dimensional system is concerned may interfere with theillusion of reality. As has been noted, it is not necessary for suchdiscrepancies to be so pronounced as to be consciously observed byviewers in order to be of potential concern. Although much of humanvision may be handled without conscious control or attention (andanomalies may not be immediately apparent at a conscious level), anon-specific sense that “this isn't right” may be problematic even if aviewer is unable to articulate or particularly identify what isn'tright.

In addition, if a three dimensional display is intended to provideinformation that aligns or interacts with a physical environment forexample, augmented reality information overlaid onto the physical world,as might be viewed using a see-through display even relatively minordifferences between perceived position and ideal position may besignificant. For instance, an outline intended to highlight a real-worldfeature such as a street sign may, to the viewer, appear to be slightlymisplaced. Besides being a potential cosmetic nuisance, such errors maycause operational problems, for example a misplaced outline mightobscure the very sign that the outline was intended to highlight.Likewise, differences in where a viewer perceives a cursor to be andwhere a controlling processor treats that cursor as being may make usingthe cursor more difficult.

Furthermore, when from the viewer's perspective displayed features arenot where those features appear to be, problems such as disorientation,dizziness, nausea, eye strain, headaches, etc. may occur. Ifsufficiently severe, such problems may limit the usefulness of a displaysystem, or even render the system effectively unusable in practice forat least some viewers.

Although variations in inter-pupillary distance are used in FIG. 1through FIG. 4 to illustrate one specific property that may cause and/orcontribute to differences between apparent and ideal positions, this isan example only, and other properties also may be significant.Individuals may exhibit variation not only in inter-pupillary distance,but also in eye alignment, in the optical architecture of the eyesthemselves, in the behavior of the eyes (i.e. saccadic motion, eyetracking, etc.), in the specifics of brain activity in interpretingimages, etc. Furthermore, although the examples in FIG. 1 through FIG. 4show only variations in deflection along one dimension (the horizontal)for purposes of simplicity, and also only show deflections that aresimilar for both eyes, individuals may perceive a target as being offsetvertically, offset to different degrees and/or in different directionswith both eyes, etc.

Furthermore, features that may have no immediately apparent connectionto eyesight may affect the apparent position of an entity displayed inthree dimensional space. For example, consider an arrangement whereinthe entity is displayed using a near-eye head mounted display, such asone resembling glasses, etc. If the head mounted display rests on theviewer's nose and ears, then factors such as the size, shape, andposition of the viewer's nose and/or ears may affect where objects areoutputted relative to the viewer's eyes, and thus may affect where thoseobjects appear to be in three dimensional space.

Besides anatomical variations, variations the position of a displayrelative to the eyes of a viewer also may affect the apparent positionof entities in three dimensional space. The viewer's preferences in howto use, wear, etc. a display system (such as wearing a head mounteddisplay high on the nose or low on the nose) thus may affect theapparent position of displayed information. For example, a viewer maywear a near-eye head mounted display high up on the bridge of the nose,or farther out near the tip of the nose. This difference in distancebetween eyes and displays may produce variations in sight lines, thuscontributing to differences in sight lines (and thus apparent distances)even for the same viewer looking at the same content; thus, evenshifting how a viewer wears or otherwise utilizes a head mounted displayfrom day to day (or moment to moment) may affect the apparent positionof content being displayed.

Also, injury and/or illness may produce variations in anatomy, preferredviewing position, etc., that are outside the typical range of healthyindividuals.

In principle, a sufficiently detailed optical model for an individualmay address such variations, for example by enabling accurate predictionof where that individual will see a target displayed with a threedimensional system. For instance, direct measurement of an individual'soptical parameters, for example inter-pupillary distance, might beundertaken. However, such measurements may require some degree of skillto execute properly, may require specialized equipment, may betime-consuming, etc. In addition, a comprehensive list of factors thatpotentially could affect depth perception may not necessarily bewell-defined, and the number of relevant factors also may be very large.A significant amount of time and skilled labor might be required togather and process such data.

By contrast, in the present invention a determination may be made as towhere a viewer in fact sees a target, as opposed to where the viewer isexpected to see the target in principle, without necessarily requiringor utilizing detailed measurement or optical modeling.

With reference now to FIG. 5, an example method for calibrating adisplay system for an individual user (and/or for an individual set ofconditions for a user, etc.) is shown therein in flow-chart form.

In the method shown in FIG. 5 an entity is outputted 544 at an idealposition. The ideal position is a location in three dimensional space atwhich it is expected the entity will be perceived by a viewer. Forexample, the entity may be outputted 544 in left and right stereodisplay screens with a position on each such screen as would ideallycorrespond with an apparent depth of (for example) 50 centimeters,centered between the viewer's eyes and in a horizontal plane alignedwith the viewer's eyes. However, this is an example only, and otherideal positions may be equally suitable.

Typically a single three-dimensional ideal position may be used todetermine corresponding two-dimensional positions for left and righteyes. For example, for a known position in three-dimensional space,ideal parallax and ideal positions in two-dimensional space may becalculated geometrically (though perhaps making assumptions regardingoptical baseline/inter-pupillary distance and other parameters, which asalready noted may vary from one individual to another).

The present invention is not particularly limited in the manner by whichthe ideal position is determined. Typically, though not necessarily, theideal position may be determined for some standardized modelrepresentative of average optical characteristics.

Continuing in FIG. 5, a viewer indication of an apparent position issensed 546. That is, the viewer indicates in some fashion where theentity appears, based on the specific properties of his or her eyes, theparticulars of the display configuration, etc., and that indication isin some manner determined. It is emphasized that the viewer is notrequired to know or understand the parameters affecting the apparentposition; the viewer is required only to indicate where the entityappears to him/her.

Typically though not necessarily, a viewer may indicate the apparentposition of the entity by aligning an end-effector such as a fingertipwith the entity. More colloquially, the viewer may point to where theentity appears to him or her.

The indication process may be binocular or monocular (and indeed themethod overall may be referred to as binocular or monocular). That is,the viewer may indicate the apparent position of the entity, and thatapparent position may be sensed, for either both eyes at the same time(binocular) or for one eye at a time (monocular). These options arediscussed in more detail later herein.

It will be understood that whatever approach is used to sense 546 theviewer indication, that approach must either provide or allow thedetermination of depth information. However, the present invention isnot otherwise particularly limited with regard to how the viewerindication may be sensed 546. For certain embodiments, a stereoarrangement of cameras, including but not limited to digital videocameras, may be suitable. For other embodiments a depth camera might beused. Other arrangements, including but not limited to ultrasonicrange-finding, active light-pulse range-finding, millimeter radar, etc.may be equally suitable.

Moving on in FIG. 5, a difference between the ideal position and theapparent position is determined 548. As has been described, where adisplayed entity is perceived to be located in three dimensional spaceis different from where that entity “should be” perceived to be locatedunder ideal conditions. Since the ideal position is known (the entityhaving been displayed at that position) and the apparent position may bedetermined from the viewer's indication, the difference between thosetwo positions also may be determined.

The present invention is not particularly limited with regard to how thedifference is determined 548, or with regard to the manner in which thatdifference is expressed. For example, the difference might be consideredin two dimensions (e.g. if convenient for systems using two dimensionaldisplays) or in three dimensions (e.g. if convenient for a threedimension space being portrayed). The difference might be expressed andconsidered as a Cartesian coordinate set (e.g. −3 cm along the x axis,+5 cm along the y axis, −11 cm z along the z axis, etc.). Alternately,the difference might be expressed as a direction and magnitude, or inother terms. As yet another alternative the difference might beexpressed as two or more values (or vectors, etc.). For example, for astereo display system a first difference may be determined for the leftdisplay and a second difference determined for the right eye, with thecombined three-dimensional difference being a result of the twoindividual differences (the difference might be expressed as onedifference for an arrangement wherein a three dimensional position isoutputted as a combination of two dimensional positions.

Still with reference to FIG. 5, a viewer-specific adjustment isdetermined 550 for entities that are and/or will be displayed. That is,based on the difference determined in step 548 where the entity appearedto be relative to where the entity was intended to be a correction maybe determined such that entities are displayed in such fashion thatwhere those entities appear to the viewer and where those entities areintended to be seen more closely match. Stated differently, applying theadjustment to the ideal position results in an apparent (adjusted)position that is substantially similar to the (non-adjusted) idealposition.

Ideally the match between the ideal position on one hand and theapparent position combined with the adjustment on the other hand shouldbe perfect, i.e. the ideal and adjusted apparent positions areidentical. However, in practice achieving perfect matching may not befeasible, nor is an identical match necessarily required by the presentinvention. For certain embodiments it may be sufficient for the idealand adjusted apparent positions to be similar, or even merely for thedifference between the ideal and adjusted apparent positions to bereduced.

With regard to less-than-identical matches, the method as shown in FIG.5 (as well as other methods shown and described herein, unless otherwisenoted) may be used iteratively. That is, the method steps in FIG. 5 maybe repeated two or more times, potentially refining/improving theadjustment so as to improve the match between ideal and adjustedapparent positions, compensating for changes in the optical parameters(e.g. if a head mounted display shifts on a viewer's head over time),etc.

Continuing in FIG. 5, the adjustment is then applied 552 to one or moreentities being displayed. The adjustment may be applied 552 to theentity that is used to determine the adjustment itself, that is, thealready-displayed position of the entity may shift from the(“incorrect”) apparent position to a new adjusted apparent position thatmore closely matches the ideal position. Similarly, the adjustment maybe applied 552 to other entities already being displayed to the viewer.In certain embodiments the displayed positions of all or nearly allinformation being displayed may be modified by applying the adjustment552, such that the entire display might be considered to recalibrate.However, this is not required, and for certain embodiments it may beequally suitable to apply the adjustment only to entities being newlydisplayed, or otherwise limit the adjustment only to some entities whilenot applying the adjustment to other entities.

With regard collectively to FIG. 6 through FIG. 13, therein is shown asequence as may correspond to at least certain features of an examplemethod for individualized three dimensional display calibrationaccording to the present invention. However, FIG. 6 through FIG. 13illustrate such features of a method according to the present inventionconsidering perceptions and/or actions as might be seen and/or carriedout by a viewer, rather than a flow-chart of individual steps as in FIG.5.

With regard specifically to FIG. 6, for a three dimensional displaysystem such as a stereo display system a distinction may exist betweenthe ideal position for a displayed entity and the apparent position ofthat displayed entity, as described previously herein. FIG. 6 shows anexample entity outputted for viewing by a viewer at an ideal position622 in three dimensional space, the viewer of the entity beingrepresented by left and right eyes 602 and 604. The arrangement in FIG.6 may be considered as at least somewhat similar to that represented bystep 544 in FIG. 5. (For purposes of simplicity, display screens, sightlines, etc. are not illustrated in FIG. 6 or in subsequent FIG. 7through FIG. 12. However, it should be understood that in at leastcertain embodiments of the present invention, changes in apparent threedimensional position of displayed content may be implemented in whole orin part by changing the actual position of content as displayed in twodimensions, e.g. on two dimensional display screens.

Turning to FIG. 7, an example entity is again shown outputted forviewing by a viewer at an ideal position 722 in three dimensional space.However, because of the particular parameters of the viewer's eyes 702and 704, the arrangement of a display outputting the example entity 722,etc., the viewer perceives the example entity as being at an apparentposition 724, with the apparent position 724 being different from theideal position 722. As may be observed, the apparent position 724 asshown in FIG. 7 is shifted to the right of the ideal position 722 fromthe point of view of the viewer (downward as FIG. 7 is arranged), and iscloser to the viewer than the ideal position 722.

It is noted that the particular shift from the ideal position 722 of theentity to the apparent position 724 of the entity as perceived by theviewer that is shown in FIG. 7 is arbitrary, presented for purposes ofillustration. As noted previously, many factors may affect apparentposition as compared with ideal position, resulting in displacement ofvarying degree and direction. No specific arrangement of opticalparameters or other properties is proposed herein as causing thisprecise shift, nor does the present invention require determining suchcause. The present invention is not particularly limited by causes ofvisual variations, and indeed the reasons for any given visual variationneed not necessarily even be known or considered. The present inventioninstead considers the viewer's perception of where things do in factappear to be, thus a determination of where things should appear to bemay be of little consequence. Such determination is not excluded fromthe present invention, however. The displacement shown in FIG. 7 is anexample only, and the present invention is not limited only todisplacements similar to that shown, or to any particular arrangement ofoptical parameters.

Moving on to FIG. 8, an example entity is shown outputted for viewing bya viewer at an ideal position 822 in three dimensional space, along withan apparent position 824 for the entity as perceived by the viewer withhis or her eyes 802 and 804. In addition, an indication by the viewer isshown in the form of the viewer pointing with a fingertip 826 to theapparent position 824. The arrangement in FIG. 8 may be considered as atleast somewhat similar to that represented by step 546 in FIG. 5.

The use of a fingertip 826 as shown in FIG. 8 to indicate the apparentposition 824 is an example only. Other end effectors, such as a stylus,a pen, a hand with other postures, etc. may be equally suitable forindicating the apparent position 824. In addition, approaches other thanthe use of an end-effector also may be equally suitable.

Turning to FIG. 9, therein is shown an example entity outputted at anideal position 922 and an apparent position 924 for the entity as viewedby the viewer 902 and 904. In addition, a difference 928 between theideal position 922 and apparent position 924 is shown. As previouslynoted and as visible in FIG. 9, the apparent position 924 is to theright of the ideal position 922 and closer to the viewer than the idealposition 922, as perceived by the viewer. The difference 928 shown inFIG. 9 is illustrated as a vector, representing the direction andmagnitude of the shift or offset between the ideal position 922 and theapparent position 924. The arrangement in FIG. 9 may be considered as atleast somewhat similar to that represented by step 548 in FIG. 5.

Although as illustrated the difference 928 is in the form of atwo-dimensional vector, the present invention is not limited only twotwo-dimensional differences, nor is the present invention particularlylimited with regard to how the difference is determined, expressed, etc.For simplicity, a two dimensional geometric vector is shown in FIG. 9,but other arrangements may be equally suitable.

Moving on to FIG. 10, again a viewer's eyes 1002 and 1004 are shownalong with an example entity outputted at an ideal position 1022 and anapparent position 1024 for the entity. The difference 1028 between theideal and apparent positions 1022 and 1024 is also shown, again as a twodimensional vector.

In addition, FIG. 10 shows a viewer adjustment 1030. The vieweradjustment 1030 is a position adjustment such that if the ideal position1022 is adjusted with the viewer adjustment 1030, the apparent positionof that adjusted ideal position is substantially similar to the(un-adjusted) ideal position 1022. That is, the apparent position of anideal position in combination with the viewer adjustment issubstantially similar to the ideal position absent the vieweradjustment.

More colloquially, the viewer adjustment 1030 “undoes” the effects ofthe difference 1028, such that a displayed entity appears to a viewer inthe ideal position 1022. The viewer adjustment 1030 as shown is anopposing vector, in effect the opposite of the difference 1028. However,the present invention is not limited only to corrections that aregeometrically, mathematically, or otherwise opposites of theirrespective differences. So long as the adjustment produces an affectaccording to the present invention, e.g. reducing and/or eliminatingvariation between where content is and where content is viewed by aviewer to be, the difference and/or viewer adjustment may varyconsiderably, and are not necessarily required to be exact opposites.

In principle, such an adjustment may be applied to the apparent position1024 rather than the ideal position 1022. However, typically (though notnecessarily) the ideal position may be more closely defined and/orcontrolled than the apparent position 1024. For example, the idealposition 1022 may be a position as calculated within a processor, whilethe apparent position 1024 is a position as perceived by the viewer.Even when the apparent position 1024 is indicated by the viewer (asshown for example in FIG. 8) the ideal position 1022 may be known withhigher precision and/or confidence than the apparent position 1024 (e.g.due to error in the viewer indication of the apparent position 1024,uncertainty in the sensing of the viewer's indication, etc.). Thustypically the viewer adjustment 1030 is determined as applicable to theideal position 1024. However, embodiments wherein a viewer adjustment isdetermined that would be applied to the apparent position 1022 are notexcluded from the present invention.

As noted elsewhere herein, the specific parameters determining theapparent position 1022 as compared with the ideal position 1024 may bedependent upon the individual viewer. Thus, a viewer correction 1030likewise may be specific to an individual viewer. Furthermore, a viewercorrection 1030 may be specific to an individual time, individualconditions, etc.

Turning to FIG. 11, a viewer's eyes 1102 and 1104 are shown along withan example entity outputted at an ideal position 1122. A vieweradjustment 1130 is also shown. As has been described, the vieweradjustment 1130 is adapted to be combined with the ideal position 1122,resulting in an adjusted ideal position 1132.

Now with reference to FIG. 12, a viewer's eyes 1202 and 1204 are shownalong with the adjusted ideal position 1232 of an example entity. For anentity displayed in the adjusted ideal position 1232, the difference1228 again may shift the entity as viewed by the viewer to an apparentposition 1224. (The apparent position 1224 of the adjusted idealposition 1232 also may be referred to as the apparent adjusted idealposition, though for clarity the location at which a viewer perceives anentity in an adjusted ideal position 1232 is referred to herein simplyas the apparent position. It is noted that the viewer may not even beaware that an adjustment has been applied to produce an adjusted idealposition 1232, thus to the viewer the apparent position 1224 of anadjusted ideal position 1232 may be simply the apparent position 1224.)

In FIG. 13, a viewer's eyes 1302 and 1304 are again shown along with anexample entity. The example entity is shown in a position correspondingto an adjusted ideal position, as perceived by the viewer. Thus thelocation of the example entity as shown is both the ideal position 1322(as not adjusted) and the apparent position 1324 of the ideal positionas adjusted. That is, the apparent position 1322 is the same as theideal position 1324 (subsequent to application of an adjustment aspreviously described). This is an example only; as noted, perfectcorrection is not required for the present invention, thus the apparentposition 1322 and the ideal position 1324 may not necessarily be exactlycoincidental, though typically the apparent position 1322 and the idealposition 1324 may be substantially similar.

Turning now to FIG. 14, an example of a binocular method for calibratinga display system for an individual user is shown therein in flow-chartform. As noted above with regard to FIG. 5, methods according to thepresent invention may be binocular or monocular. That is, a viewer mayindicate the apparent position of an entity, and that apparent positionmay be sensed, for either both eyes at the same time (binocular) or forone eye at a time (monocular). The arrangement of FIG. 5 does notspecify either binocular or monocular method, rather being generic toboth. FIG. 14 shows an explicitly binocular method, that is, a methodwherein both of the viewer's eyes are considered together.

In the method shown in FIG. 14 an entity is outputted 1444 at an idealposition in binocular displays (e.g. left and right displays of a stereodisplay system). The entity is outputted 1444 so as to appearsubstantially simultaneously to both eyes of the viewer, such that theviewer perceives the entity and the entity's position in threedimensional space using binocular vision.

A binocular viewer indication of an apparent position is sensed 1446.That is, the viewer indicates where the entity appears in his or herbinocular vision, i.e. as the entity appears to both eyes at the sametime. Thus, a single indication identifies where the entity appears tothe viewer to both his or her left and right eyes, that singleindication is sensed 1446 for determining the apparent position asviewed by both of the user's eyes.

Typically though not necessarily, a viewer may indicate the apparentposition of the entity by aligning an end-effector such as a fingertipwith the entity, as substantially simultaneously viewed by both of theviewer's eyes.

Moving on in FIG. 14, a difference between the binocular ideal positionsand the binocular apparent position is determined 1448. As noted, only asingle three-dimensional apparent position is indicated. Typically twotwo-dimensional ideal positions are generated for binocular display, sotwo differences may be determined 1448. However, also typically the twotwo-dimensional ideal positions are derived from a singlethree-dimensional ideal position, so for certain embodiments only asingle difference may be determined, and the present invention is notparticularly limited in this regard.

A viewer-specific adjustment is determined 1450 for entities that areand/or will be displayed. That is, based on the difference determined instep 1448, a correction is determined such that where entities appear tothe viewer and where those entities are intended to appear more closelymatch, i.e. applying the adjustment to the ideal position results in anapparent position that is substantially similar to the (non-adjusted)ideal position.

Continuing in FIG. 14, the adjustment is then applied 1452 to one ormore entities being displayed. The adjustment may be applied 1452 to theentity that is used to determine the adjustment itself, or may beapplied only to other entities. With the adjustment applied 1452, asdescribed earlier the apparent position of the adjusted ideal positionshould be substantially similar or even identical to the non-adjustedideal position.

Turning now to FIG. 15, therein an example of a monocular method forcalibrating a display system for an individual user is shown therein inflow-chart form. That is, in the method of FIG. 15 each of the viewer'seyes is considered separately.

In the method shown in FIG. 15 an entity is outputted 1544A at a firstideal position in a first display. The entity as outputted 1544A at itsfirst ideal position thus is visible to a first of the viewer's eyes,but not to the second eye. Typically though not necessarily, the seconddisplay may be left blank, blacked out, partially darkened, reduced incontrast, or otherwise configured so that the viewer may more readilyconcentrate on the image in the first display without being distractedby information from the second display.

A viewer indication of the first apparent position is sensed 1546A. Thisis a monocular determination, that is, the viewer indicates where theentity appears in his or her vision for only one eye. Typically thoughnot necessarily, the viewer may indicate the first apparent position bypointing to that position with a fingertip or other end-effector, thoughother arrangements may be equally suitable.

Continuing in FIG. 15, the entity is outputted 1544B at a second idealposition in a second display. The entity as outputted 1544B at itssecond ideal position thus is visible to a second of the viewer's eyes,but not to first second eye. Typically though not necessarily, the firstdisplay may be left blank, blacked out, partially darkened, reduced incontrast, or otherwise configured so that the viewer may more readilyconcentrate on the image in the second display without being distractedby information from the first display.

A viewer indication of the second apparent position is sensed 1546B. Aswith step 1546A above, step 1546B is a monocular determination, whereinthe viewer indicates where the entity appears in his or her vision foronly one eye (the other eye than in step 1546A). Typically though notnecessarily, the viewer may indicate the second apparent position bypointing to that position with a fingertip or other end-effector, thoughother arrangements may be equally suitable.

As already noted, the present invention is not particularly limited withregard to how the viewer indicates apparent positions or how apparentpositions are sensed. With regard to monocular indications, it is alsonoted that the viewer may indicate the first and second apparentpositions differently. For example, a viewer may use one fingertip toindicate the first apparent position, and a different fingertip toindicate the second apparent position. Likewise, sensing the monocularindications 1546A and 1546B may be carried out in different ways foreach sensing step 1546A and 1546B.

Also, for indication of a monocular first or second apparent position asin steps 1544A and 1544B, the viewer indications typically may beexecuted as two-dimensional position indications. For example, for astereo display system using essentially two-dimensional displays, foreach eye the output visible to the viewer is substantially twodimensional (even though the combined screens may produce athree-dimensional stereo effect) so that each indication of apparentposition likewise may be substantially or entirely two-dimensional.Thus, at what distance the viewer indicates the first or second apparentposition, e.g. by pointing with a fingertip, may for certain embodimentsbe ignored while still providing an ability to determine athree-dimensional apparent position by sensing apparent positions forboth eyes (though one eye at a time rather than together). In otherwords, the viewer may not be required to accurately indicate a distance(or need not indicate a distance at all) for each apparent position, andthe distance may need not need to be sensed accurately (or at all) whensensing each apparent position 1546A and 1546B.

Still with reference to FIG. 15, although for simplicity the entityoutputted in steps 1544A and 1544B is referred to in the singular, asone entity, what is displayed in the first and second displays is notnecessarily required to be identical for all embodiments of the presentinvention. For example, if the entity is three-dimensional, theperspective of that entity as shown in the first and second displays maybe slightly different because displays are at slightly differentpositions. In such instance the appearance of the entity, as well as theposition of the entity, may be slightly different for the first andsecond displays (and thus for the viewer's left and right eyes).

In addition, the present invention is not limited only to the use of asingle entity for both the first and second displays. For certainembodiments, it may be equally suitable to display a first entity at afirst ideal position in the first display, and a second (perhapsentirely different) entity at a second position in the second display.

Moving on in FIG. 15, a first difference is determined 1548A between thefirst ideal position and the first apparent position. Similarly, asecond difference is determined 1548B between the second ideal positionand the second apparent position.

A viewer-specific adjustment is determined 1550 for entities that areand/or will be displayed. That is, based on the first and seconddifferences determined in steps 1548A and 1548B, a correction isdetermined such that where entities appear to the viewer and where thoseentities are intended to appear more closely match, i.e. applying theadjustment to the ideal position results in an apparent position that issubstantially similar to the (non-adjusted) ideal position.

Although first and second ideal positions are outputted 1544A and 1544B,typically (though not necessarily) those first and second idealpositions are two dimensional positions that, in combination with abaseline between eyes (e.g. the inter-pupillary distance) triangulate asingle three-dimensional ideal position. Thus, although there are alsofirst and second apparent positions sensed in steps 1546A and 1546B, andfirst and second differences 1548A and 1548B, there may for at leastcertain embodiments be only a single adjustment. That is, thethree-dimensional ideal position may be adjusted, rather than therebeing separate adjustments for the first and second (two-dimensional)ideal positions. (If the three-dimensional ideal position is adjusted,then two-dimensional first and second ideal positions derived therefromlikewise may be adjusted, even if not directly.)

Moving on in FIG. 15, the adjustment is then applied 1552 to thepositions of one or more entities being displayed. The adjustment may beapplied 1552 to the entity that is used to determine the adjustmentitself, or may be applied only to other entities. With the adjustmentapplied 1552, as described earlier the apparent position of the adjustedideal position should be substantially similar or even identical to thenon-adjusted ideal position. Likewise, first and second apparentpositions (two-dimensional) of entities derived from adjusted idealpositions (three-dimensional) also should be substantially similar oridentical to their respective non-adjusted ideal positions.

As noted above with regard to FIG. 15, for at least certain embodimentsonly one three-dimensional adjustment may be determined (as in step1550) and applied (as in step 1552). However, for at least certainembodiments, individual adjustments may be made for each of first andsecond ideal positions. That is, individual two-dimensional displayedpositions on first and second displays may be adjusted individually,rather than adjusting a three-dimensional position (and relying on thatadjustment to apply to individual two-dimensional first and second idealpositions derived therefrom). Such an arrangement is shown in FIG. 16.

With regard to FIG. 16, up to this point it has been assumed forsimplicity that relevant positions and/or position information (e.g. athree-dimensional ideal position, two-dimensional first and second idealpositions, etc.) may already be available in some form. However, in FIG.16 the establishing of such ideal positions is made explicit.

In FIG. 16 another example of a monocular method for calibrating adisplay system for an individual user is shown therein in flow-chartform.

A three-dimensional position is established 1640, e.g. for some virtualor augmented entity such as a test marker, etc., though otherarrangements may be equally suitable.

The concept of establishing a position is to be considered broadly withregard to the present invention. It is noted that to “establish”something may, depending on particulars, refer to either or both thecreation of something new (e.g. establishing a business, wherein a newbusiness is created) and the determination of a condition that alreadyexists (e.g. establishing the whereabouts of a person, wherein thelocation of a person who is already present at that location isdiscovered, received from another source, etc.). Similarly, establishinga position may encompass several potential approaches, including but notlimited to the following.

Establishing a position may include measuring, approximating, etc. theposition from existing positions, spatial relationships, etc., includingbut not limited to positions of physical objects, virtual objects,augmented objects, etc.

Establishing a position also may include creating or calculating theposition without necessarily having a connection to existing spatialarrangements, e.g. a processor may execute instructions so as to computeor create a position in some fashion, whether from existing data, userinputs, internal algorithms, etc.

Establishing a position additionally may include selecting apreviously-existing position, for example by reading positioninformation from a data store, downloading a position from acommunication link, or otherwise obtaining a position that alreadyexists substantially in a form as to be utilized by some embodiment ofthe present invention.

The present invention is not particularly limited insofar as how aposition may be established.

Continuing in FIG. 16, a first ideal position is established 1642A for afirst display. A second ideal position also is established 1642B for asecond display. First and second ideal positions have been describedpreviously herein. As also described previously with regard to step1640, the term “established” when applied to establishing first andsecond ideal positions should be understood broadly; first and secondideal positions may be received, calculated, created, etc., and thepresent invention is not particularly limited with regard to how thefirst and second ideal positions may be established 1642A and 1642B.

Continuing in FIG. 16, an entity is outputted 1644A at a first idealposition in a first display. The entity as outputted 1644A at its firstideal position thus is visible to a first of the viewer's eyes, but notto the second eye. Typically though not necessarily, the second displaymay be left blank, blacked out, partially darkened, reduced in contrast,or otherwise configured so that the viewer may more readily concentrateon the image in the first display without being distracted byinformation from the second display.

A viewer indication of the first apparent position is sensed 1646A. Thisis a monocular determination, that is, the viewer indicates where theentity appears in his or her vision for only one eye. Typically thoughnot necessarily, the viewer may indicate the first apparent position bypointing to that position with a fingertip or other end-effector, thoughother arrangements may be equally suitable.

The entity is outputted 1644B at a second ideal position in a seconddisplay. The entity as outputted 1644B at its second ideal position thusis visible to a second of the viewer's eyes, but not to first secondeye. Typically though not necessarily, the first display may be leftblank, blacked out, partially darkened, reduced in contrast, orotherwise configured so that the viewer may more readily concentrate onthe image in the second display without being distracted by informationfrom the first display.

A viewer indication of the second apparent position is sensed 1646B. Aswith step 1646A above, step 1646B is a monocular determination, whereinthe viewer indicates where the entity appears in his or her vision foronly one eye (the other eye than in step 1646A). Typically though notnecessarily, the viewer may indicate the second apparent position bypointing to that position with a fingertip or other end-effector, thoughother arrangements may be equally suitable.

Moving on in FIG. 16, a first difference is determined 1648A between thefirst ideal position and the first apparent position. Similarly, asecond difference is determined 1648B between the second ideal positionand the second apparent position.

A viewer-specific first adjustment is determined 1650A for entities thatare and/or will be outputted to the first display. That is, based on thefirst difference determined in step 1648A, a correction is determinedsuch that where entities appear to the viewer and where those entitiesare intended to appear more closely match, i.e. applying the adjustmentto the ideal position results in an apparent position that issubstantially similar to the (non-adjusted) ideal position.

A viewer-specific second adjustment likewise is determined 1650B forentities that are and/or will be outputted to the second display.

The first adjustment is then applied 1652A to the positions of one ormore entities being outputted to the first display. The secondadjustment is similarly applied 1652B to the positions of one or moreentities that are to be outputted to the second display.

Still with reference to FIG. 16, an entity is outputted to the firstdisplay 1654A, with the first adjustment applied thereto. An entity isoutputted to the second display 1654B, with the second adjustmentapplied thereto. Typically for a three dimensional display such as astereo display, the same entity is outputted to the first and seconddisplays 1654A and 1654B so as to produce the effect of the entity beingpositioned within three dimensional space. Thus although two outputs aregenerated (one to the first display and one to the second), the outputscollectively may be considered and/or processed as a single entity.However, arrangements wherein the outputs constitute a first entityoutputted to the first display 1654A and a second entity outputted tothe second display may be equally suitable.

The outputted entity or entities may include the original entity orentities for which the viewer indicated an apparent position. In suchcase the position of that entity as outputted may be adjusted.Alternately, the output adjustment may be applied to one or moreadditional entities without necessarily being applied to the originalentity. That is, considering the original entity to be the first entity,the first entity may be outputted, a second entity may be outputted,and/or both the first entity and a second entity may be outputted.

As has been noted, embodiments of the present invention may be binocularand/or monocular. Binocular and monocular approaches each may haveadvantages for at least certain embodiments of the present invention.

For example, a binocular approach requires only one sighting andindication on the part of the viewer. Thus, from the perspective of aviewer a binocular approach may require less effort than a monocularapproach (which requires two sightings and indications), may be lessdisruptive of ongoing work, etc. In addition, a binocular approach isdirect, in the sense that the viewer identifies a three-dimensionalpoint by making a three-dimensional indication (i.e. with both eyes atonce).

Conversely, a monocular approach may require less precision on the partof the viewer. For a monocular approach, the viewer is not required toaccurately perceive depth (which may for certain viewers beproblematic); rather than indicating with a single action the apparentposition of a target in three-dimensional space, the viewer may indicatetwo points in two-dimensional space. A monocular approach thus may, forat least certain embodiments, provide a determination of apparentposition that is more accurate and/or more precise than a binocularapproach.

Likewise, embodiments of the present invention may be explicit orimplicit. That is, calibration according to the present invention mayfor certain embodiments be accomplished as steps that are identified tothe viewer as being a calibration (explicit), and/or may be accomplishedas steps that are not so identified.

Again, each approach may have advantages for certain embodiments. Forexample, explicit calibration may be useful for an arrangement whereinthe degree of calibration (e.g. how much offset may be expected to existbetween ideal and apparent positions) may reasonably be anticipated tobe large, such as when first calibrating a system to a new viewer whoseparameters are unknown. The viewer, being aware that he or she isexecuting calibration, may take special care to be accurate inindicating apparent positions, may be aware of and/or accommodate largeoffsets, may accept several repetitions of the method to comprehensivelycalibrate the system, etc.

Conversely, implicit calibration may for at least certain embodiments bemade transparent to a viewer. For example, if a system is protected witha PIN or password, a viewer might be expected to “type” the PIN orpassword by aligning his or her fingertip with targets representingletters and numbers. As another example, a system might be activated bydragging an icon from an initial position to a specified position.Either arrangement (or many others within the scope of the presentinvention) provide opportunities to determine apparent positions ascompared with ideal positions. Calibration thus may be integrated withsome other useful function wherein the viewer communicates with theprocessor (e.g. entering a password), so that no additional time orconcentration is required from the viewer beyond that of thecommunication itself. Such calibrations may be effectively invisible tothe viewer, with no requirement for the viewer to dedicate additionaltime or effort to calibration.

In particular, it is noted that such implicit and/or transparentcalibration may be executed repeatedly, and/or on an ongoing basis. Ashas been noted, the methods as described herein may be repeated. Themethods of the present invention may indeed be carried out as part ofthe normal ongoing function of a system, for example when a viewerinteracts with the system by gestures or other position indications, theoffsets between apparent and ideal positions may be updated, refined,etc. Potentially every such interaction may be utilized as part of anongoing, self-adjusting calibration process.

However, such arrangements are examples only. The present invention isnot particularly limited with regard to monocular vs. binocularapproaches (or other suitable approaches), explicit vs. implicitapproaches (or other suitable options), repetition or lack ofrepetition, etc.

Having described several examples of methods for individualized threedimensional display calibration according to the present invention, atthis time it may be useful to note that the present invention does notrequire measurement of the viewer's eyes, the optical propertiesthereof, etc. For example, the present invention does not requiremeasurement or even approximation of the inter-pupillary distance of aparticular viewer. Rather, the present invention relies upon theviewer's perception of where targets are positioned. While geometricand/or optical factors (and/or other parameters) such as inter-pupillarydistance may be a contributing factor in where the viewer perceivesvarious targets, measurement of such parameters is not required (thoughmeasurement also is not prohibited). As has been noted earlier hereincomprehensively identifying and measuring all potentially relevantfactors for the optical system of a viewer, or even identifying andmeasuring the most significant relevant factors, may be problematic. Bycontrast, the present invention does not require identification ormeasuring of such factors. The present invention takes advantage of theviewer indicating what he or she does in fact see, rather thannecessarily analyzing or predicting (for example based on the viewer'sparticular optical parameters) what the viewer ought to see.

Thus, although optical measurement are not prohibited in the presentinvention, neither are optical measurements required in executing anindividualized three dimensional display calibration according to thepresent invention.

Turning now to FIG. 17, therein another example of a monocular methodfor calibrating a display system for an individual user is shown thereinin flow-chart form. Except where otherwise indicated herein, the presentinvention is not particularly limited with regard to how steps thereofare carried out. However, for clarity FIG. 17 provides examples ofcertain approaches as may be suitable for at least some embodiments ofthe present invention. These approaches are examples only, and otherapproaches may be equally suitable, With reference to certain stepsherein, alternatives also may be noted for explanatory purposes. Wherealternatives are provided, it should be understood that description ofsome alternatives does not necessarily imply that other alternatives areunsuitable.

Furthermore, although FIG. 17 refers to a specific example of amonocular method. according to the present invention, it may beunderstood that such concrete examples as shown in FIG. 17 may similarlyapply to at least certain binocular and/or other arrangements accordingto the present invention.

The example of FIG. 17 refers to a head mounted display (or HMD). A headmounted display at least somewhat similar to that referenced in FIG. 17is shown in FIG. 20 and described subsequently with regard thereto, withadditional description of elements thereof (or at least suitabletherefor) also shown in and described with regard to FIGS. 18, 19, and21.

In the example method of FIG. 17, a three-dimensional position for avisual calibration target is determined 1740 in the processor of a headmounted display. Given such an arrangement, the position might be readfrom a data store, determined computationally using executableinstructions instantiated on the processor, etc.

Although for the example of FIG. 17 the position is determined in theprocessor, e.g. through access to and/or manipulation of data, for atleast some embodiments such positions may be determined partially and/orentirely with reference to physical objects. For example, a position ofan augmented reality calibration target may be disposed with referenceto some real-world object within the viewer's field of view, e.g.overlaid onto a physical object (whether at the same or a differentapparent distance from the viewer), positioned adjacent a physicalobject, etc. In such case, sensor data such as images and/or otherfactors may be considered in making the determination 1740.

Continuing in FIG. 17, a first ideal position for the target in a firststereo display (e.g. left or right) of the HMD is determined 1742A inthe HMD processor. A second ideal position for the target in a secondstereo display of the HMD also is established 1742B in the HMDprocessor.

The target is outputted 1744A from the HMD processor at a first idealposition in a first stereo display of the HMD. The target as outputted1744A at its first ideal position thus is visible to a first of theviewer's eyes, but not to the second eye. A viewer indication of thefirst apparent position is sensed 1746A with a depth imager on the HMD.For example, an image may be captured 1746A using a depth cameraintegrated into the HMD, showing the viewer's fingertip pointing towhere the viewer sees the target in the first stereo display of the HMD.That linage then may be evaluated (e.g. in the HMD processor) so as todetermine the position of the viewer's fingertip in a format suitablefor computation by the HMD processor in subsequent steps.

Moving on in FIG. 17, the target is outputted 1744B from the HMDprocessor at a second ideal position in a second stereo display of theHMD. The target as outputted 1744B at its second ideal position thus isvisible to a second of the viewer's eyes, but not to the first eye. Aviewer indication of the second apparent position is sensed 1746B with adepth imager on the HMD (potentially though not necessarily the samedepth imager or other sensor used in step 1746A).

In the HMD processor, a first difference is determined 1748A between thefirst ideal position and the first apparent position. Also in theprocessor, a second difference is determined 1748B between the secondideal position and the second apparent position. Steps 1748A and 1748Bmay, for example, be performed computationally through the use ofexecutable instructions instantiated on the processor.

Again in the HMD processor, a first adjustment is determined 1750A forthe target as viewed in the first stereo display of the HMD,representing a correction such that where entities appear to the viewerand where those entities are intended to appear more closely match. Thisfirst adjustment also may be utilized for other entities that are and/orwill be outputted to the first display. A second adjustment also isdetermined 1750B for the target as viewed in the second stereo displayof the HMD.

The first adjustment is then applied 1752A in the HMD processor (e.g.computationally through the use of executable instructions instantiatedthereon) to the position of the target as outputted to the first stereodisplay of the HMD. The first adjustment may be applied similarly toother entities being and/or to be outputted to the first display. Thesecond adjustment also is applied 1752B in the HMD processor to theposition of the target as outputted to the second stereo display of theHMD; the second adjustment likewise may be applied to other entitiesbeing and/or to be outputted to the second display.

Still with reference to FIG. 17, the target is outputted to the firststereo HMD display 1754A, with the first adjustment applied thereto. Thetarget is outputted to the second stereo HMD display 1754B, with thesecond adjustment applied thereto. For the example of FIG. 17, then, thetarget as viewed by the viewer may appear to move from an initial idealposition to a subsequent adjusted position.

Turning now to FIG. 18, a schematic diagram of an embodiment of anapparatus 1860 for individualized three dimensional display calibrationis shown therein.

The apparatus 1860 includes a processor 1862 adapted for executingexecutable instructions. The invention is not particularly limited withregard to the choice of processor 1862. Suitable data processors 1862include but are not limited to digital electronic microprocessors.Although the processor 1862 is referred to in at least some placesherein as a self-contained physical device for purposes of clarity, thisis not required, and other arrangements may be suitable. For example,the processor 1862 may constitute two or more physical processorsworking cooperatively, a processing capability in a network without awell-defined physical form, etc.

The apparatus 1860 also includes a display 1864 in communication withthe processor 1862. The display 1864 is adapted to output at least afirst entity at an ideal position. Typically though not necessarily thedisplay 1864 may be adapted to output many entities, and/or to outputsome or all of an augmented reality and/or virtual reality environment,e.g. as imagery to a user. However, this is an example only, and otherarrangements may be equally suitable.

As illustrated, the display 1864 is a stereo display, with left andright screens adapted to output to the left and right eyes of a viewer.However this also is an example only. The present invention is notparticularly limited with regard to the type of display 1864. Typically,although not necessarily, the display 1864 may be a visual display. Inaddition, the display 1864 may be a transparent or see-through display,wherein augmented reality information may be displayed in combinationwith real world information that may be viewed through the transparentdisplay (though this does not necessarily exclude such a displayobstructing or overwriting the entire field of view so as to displayvirtual reality information without passing real world informationtherethrough). Alternately, the display 1864 may be a “virtually”transparent display, wherein augmented reality information is visiblethereon along with information from the real world that is activelydisplayed (e.g. having been captured with cameras) without the displaynecessarily being physically transparent.

The present invention is not particularly limited with regard to thedisplay 1864. A range of devices may be suitable for use as the display1864, including but not limited to light emitting diodes (LED), organiclight emitting diodes (OLED), plasma screen panels (PDP), liquid crystaldisplays (LCD), etc. Likewise, the use of projected or transmitteddisplays, where the viewed surface is essentially a passive screen foran image projected or otherwise transmitted after being generatedelsewhere, may also be suitable. Other arrangements including but notlimited to systems that display images directly onto a user's eyes alsomay be equally suitable. Either digital or analog display technologiesmay be suitable. Furthermore, as noted the present invention is notlimited only to the use of visual displays as a display 1864.

Still with reference to FIG. 18, the apparatus 1860 includes at leastone sensor 1866 in communication with the processor 1862. The sensor1866 is adapted to sense a viewer indication of the apparent position ofthe first entity. Typically though not necessarily, the sensor 1866 alsomay sense additional information, for example imaging some or all of theregion surrounding the apparatus 1860. As a more concrete example, foran arrangement wherein the apparatus 1860 outputs to a viewer anaugmented reality environment, the sensor 1866 may capture some or allof the real world information that is displayed and augmented within theaugmented reality environment, in addition to that same sensor 1866sensing viewer indications of ideal positions.

As shown, the sensor 1866 in FIG. 18 is illustrated as an imager such asa digital video camera, but this is an example only. The presentinvention is not particularly limited with regard to the sensor 1866,nor with how the sensor 1866 senses indications of ideal positionsand/or other information. Suitable sensors 1866 may include but are notlimited to imagers, depth sensors, structured light sensors,time-of-flight sensors, and ultrasonic sensors.

The apparatus 1860 includes several elements shown to be instantiated onthe processor 1862. The aforementioned elements include an offsetdeterminer 1874, an adjustment determiner 1876, and an adjustmentapplier 1878. Typically, the offset determiner 1874, adjustmentdeterminer 1876, and adjustment applier 1878 include executableinstructions and/or data, e.g. instantiated on the processor 1862, andin at least some embodiments the offset determiner 1874, adjustmentdeterminer 1876, and adjustment applier 1878 may be composed exclusivelyof executable instructions and/or data, but this is an example only.

However, for purposes of clarity for the example embodiment shown inFIG. 18, the offset determiner 1874, adjustment determiner 1876, andadjustment applier 1878 may be referred to in at least some placesherein as being composed of executable instructions and/or datainstantiated on the processor 1862.

It is noted further that although the offset determiner 1874, adjustmentdeterminer 1876, and adjustment applier 1878 are shown and describedherein as being separate elements, this is done for clarity and shouldnot be taken to limit the present invention. For at least someembodiments, one or more of the offset determiner 1874, adjustmentdeterminer 1876, and adjustment applies 1878 may be combined with oneanother, and/or may be incorporated into some larger construct, e.g. asingle program performing all functions thereof, a general operatingsystem, etc. Furthermore, any or all of the offset determiner 1874adjustment determiner 1876, and adjustment applier 1878 may besubdivided.

In the arrangement of FIG. 18, the offset determiner 1874 is adapted todetermine an offset between an ideal position (typically though notnecessarily provided by the processor 1862) and an apparent position (assensed by the sensor 1866 from the viewer's indication). Typicallythough not necessarily the determination of offset is an arithmeticaland/or geometric measurement and/or computation. Also typically theoffset determiner 1874 may determine offsets in two dimensions and/or inthree dimensions, though this may depend on the particulars of a givenembodiment (e.g. a stereo display system might determine a single threedimensional position or a two dimensional position for each of thestereo displays), and other arrangements may be equally suitable. Idealpositions and apparent positions have been described previously herein.

The adjustment determiner 1876 is adapted to determine an outputadjustment for the ideal position, such that, the apparent position ofthe ideal position with the adjustment is substantially similar to theideal position absent said adjustment. Typically though not necessarilythe determination of adjustment is an arithmetical and/or geometricmeasurement and/or computation, based on the ideal position and offset(as determined by the offset determiner 1874). The adjustment determiner1876, like the offset determiner 1874, may operate in two dimensions, inthree dimensions, etc. Output adjustments have been described previouslyherein.

The adjustment applier 1878 is adapted to apply the output adjustment(as determined by the adjustment determiner 1876) to one or moreentities to be outputted by the display 1864. This may include theoriginal target, i.e. the entity for which the viewer indicated anapparent position, in which case the position of that entity asoutputted may be adjusted. Alternately, the output adjustment may beapplied to one or more additional entities without necessarily beingapplied to the target.

Turning to FIG. 19, therein is shown a schematic diagram of anotherembodiment of an apparatus 1960 for individualized three dimensionaldisplay calibration. The apparatus in FIG. 19 corresponds at leastsomewhat with the method shown in FIG. 16, in that the arrangement ofFIG. 19 includes therein additional elements for establishing idealpositions as is shown in FIG. 16. In addition, the arrangement of FIG.19 is adapted to address two separate two dimensional positions ratherthan one three dimensional position. Typically for a display such as astereo display a pair of two dimensional positions are functionallyequivalent to a single three dimensional position, but for certainembodiments it may be more convenient (e.g. computationally) to considertwo dimensional positions rather than a three dimensional position forat least certain steps. Other arrangements also may be equally suitable.

The apparatus 1960 shown in FIG. 19 includes a processor 1962 and anoutputter 1964. The apparatus also includes first and second sensors1966A and 1966B in a stereo configuration, though this configuration isan example only and other arrangements may be equally suitable. Theapparatus 1960 also includes an offset determiner 1974, an adjustmentdeterminer 1976, and an adjustment applier 1978 instantiated on theprocessor 1962. These elements are at least somewhat similar to elementsin FIG. 18 as already described.

In addition, the apparatus 1960 of FIG. 19 includes several furtherelements also shown to be instantiated on the processor 1962: a positionestablisher 1968, a stereo position determiner 1970, and an outputter1972. Typically the position establisher 1968, stereo positiondeterminer 1970, and outputter 1972 include executable instructionsand/or data, e.g. instantiated on the processor 1962, and in at leastsome embodiments the position establisher 1968, stereo positiondeterminer 1970, and outputter 1972 may be composed exclusively ofexecutable instructions and/or data, but this is an example only.However, for purposes of clarity for the example embodiment shown inFIG. 19, the position establisher 1968, stereo position determiner 1970,and outputter 1972 may be referred to in at least some places herein asbeing composed of executable instructions and/or data instantiated onthe processor 1962.

As noted with regard to elements 1874, 1876, and 1878 in FIG. 18, one ormore of the position establisher 1968, stereo position determiner 1970,and outputter 1972 in FIG. 19 may be combined with one another, may beincorporated into some larger construct, and/or may be subdivided.

The position establisher 1968 is adapted to establish a substantiallythree-dimensional combined ideal position. Ideal positions andestablishing positions have been described previously herein. Typically,the position establisher 1968 may establish a three dimensional idealposition by calculating that position, reading the position from a datastore, receiving the position from a program such as one creating oroperating a virtual or augmented reality environment, etc. However,other arrangements may be equally suitable.

The stereo position determiner 1970 is adapted to two dimensional or atleast substantially two-dimensional first and second ideal positions foroutput to the stereo display 1964, for example a first ideal positionfor the left portion of the display 1964 (to be viewed by the left eye)and a second ideal position for the right portion of the display 1964(to be viewed by the right eye), such that those two-dimensional firstand second ideal positions at least substantially correspond with thethree dimensional ideal position. That is, the first and second twodimensional ideal positions are such that, when viewed by a viewer, thefirst and second two dimensional stereo positions collectively appear asa single three dimensional position (i.e. the three dimensional combinedideal position).

The outputter 1972 is adapted to output a first target substantially tothe first two dimensional ideal position in the display 1964, and asecond target substantially to the second two dimensional ideal positionin the display 1964. It is noted that although the stereo display 1964in FIG. 19 is shown as an integral unit with two separate screens, forother embodiments it may be equally suitable to use two separatedisplays, to treat a single display logically as two units, to use asingle screen with left and right images thereon, etc.

Still with reference to FIG. 19, an offset determiner 1974 isinstantiated on the processor 1962. The offset determiner 1962 isadapted to determine offsets between the first and second twodimensional ideal positions and first and second apparent positionsindicated by the viewer and sensed by the sensors 1966A and 1966B.

An adjustment determiner 1976 is instantiated on the processor 1962. Theadjustment determiner 1976 is adapted to determine output adjustmentsfor the first and second two dimensional ideal positions, such that theapparent positions of the first and second two dimensional idealpositions with their respective first and second adjustments aresubstantially similar to the first and second two dimensional idealpositions, respectively.

An adjustment applier 1978 also is instantiated on the processor 1962.The adjustment applier 1978 is adapted to apply the first and secondoutput adjustments (as determined by the adjustment determiner 1976) toone or more entities to be outputted by the display 1964.

Turning now to FIG. 20, the present invention is not particularlylimited with regard to form, and may be disposed on and/or incorporatedinto many shapes and/or other devices. Suitable configurations includebut are not limited to the example shown in FIG. 20, wherein the presentinvention is illustrated in the form of a head mounted displayresembling a pair of glasses.

As shown in FIG. 20, the example embodiment of the apparatus 2060therein includes a body 2080 having a form similar to a pair of glasses,and adapted to be worn in a similar fashion. A processor 2062 adaptedfor executing executable instructions is disposed on the body 2080.Although not visible as distinct entities, the processor 2062 maysupport elements such as a position establisher, a stereo positiondeterminer, an outputter, an offset determiner, an adjustmentdeterminer, and an adjustment applier, e.g. in the form of executableinstructions and/or data instantiated on the processor 2062.

The apparatus 2060 also includes sensors 2066A and 2066B disposed on thebody 2080, illustrated in FIG. 20 as imagers in a stereo configuration,though these are examples only. The apparatus 2080 further includesdisplays 2064A and 2064B disposed on the body 2060, illustrated as leftand right visual displays in a stereo configuration.

It is noted that in the configuration shown, the body 2060 is configuredand sensors 2066A and 2066B are disposed thereon such that when the body2060 is worn by a viewer, the sensors 2066A and 2066B would besubstantially aligned with the lines of sight of the viewer's eyes, andcould potentially encompass fields of view at least somewhat comparableto those of the viewer's eyes, assuming sensors 2066A and 2066B withfields of view similar in extent to those of the viewer. Such anarrangement might for example be suitable for receiving a viewerindication of an apparent position. Similarly, in the configurationshown the body 2060 is configured and the displays 2064A and 2064B aredisposed thereon such that when the body 2060 is worn by a viewer, thedisplays 2064A and 2064B would be proximate to and substantially infront of the viewer's eyes, for example so as to output entities to aviewer.

However, it is emphasized that the arrangement in FIG. 20 is an exampleonly, and that other arrangements may be equally suitable.

FIG. 21 is a block diagram of an apparatus that may perform variousoperations, and store various information generated and/or used by suchoperations, according to an embodiment of the disclosed technique. Theapparatus may represent any computer or processing system describedherein. The processing system 2190 is a hardware device on which any ofthe other entities, components, or services depicted in the examples ofFIG. 1 through FIG. 20 (and any other components described in thisspecification) may be implemented. The processing system 2190 includesone or more processors 2191 and memory 2192 coupled to an interconnect2193. The interconnect 2193 is shown in FIG. 20 as an abstraction thatrepresents any one or more separate physical buses, point to pointconnections, or both connected by appropriate bridges, adapters, orcontrollers. The interconnect 2193, therefore, may include, for example,a system bus, a Peripheral Component Interconnect (PCI) bus orPCI-Express bus, a HyperTransport or industry standard architecture(ISA) bus, a small computer system interface (SCSI) bus, a universalserial bus (USB), IIC (I2C) bus, or an Institute of Electrical andElectronics Engineers (IEEE) standard 1394 bus, also called “Firewire”.

The processor(s) 2191 is/are the central processing unit of theprocessing system 2190 and, thus, control the overall operation of theprocessing system 2190. In certain embodiments, the processor(s) 2191accomplish this by executing software or firmware stored in memory 2192.The processor(s) 2191 may be, or may include, one or more programmablegeneral-purpose or special-purpose microprocessors, digital signalprocessors (DSPs), programmable controllers, application specificintegrated circuits (ASICs), programmable logic devices (PLDs), trustedplatform modules (TPMs), or the like, or a combination of such devices.

The memory 2192 is or includes the main memory of the processing system2190. The memory 2192 represents any form of random access memory (RAM),read-only memory (ROM), flash memory, or the like, or a combination ofsuch devices. In use, the memory 2192 may contain a code. In oneembodiment, the code includes a general programming module configured torecognize the general-purpose program received via the computer businterface, and prepare the general-purpose program for execution at theprocessor. In another embodiment, the general programming module may beimplemented using hardware circuitry such as ASICs, PLDs, orfield-programmable gate arrays (FPGAs).

The network storage adapter 2194, a storage devices) 2195, and I/Odevice(s) 2196, are also connected to the processor(s) 2191 through theinterconnect 2193. The network adapter 2194 provides the processingsystem 2090 with the ability to communicate with remote devices over anetwork and may be, for example, an Ethernet adapter or Fibre Channeladapter. The network adapter 2194 may also provide the processing system2190 with the ability to communicate with other computers within thecluster. In some embodiments, the processing system 2190 may use morethan one network adapter to deal with the communications within andoutside of the cluster separately.

The I/O device(s) 2196 can include, for example, a keyboard, a mouse orother pointing device, disk drives, printers, a scanner, and other inputand/or output devices, including a display device. The I/O device(s)2196 also may include, for example, cameras and/or other imagers adaptedto accept visual input including but not limited to postures and/orgestures. The display device may include, for example, a cathode raytube (CRT), liquid crystal display (LCD), or some other applicable knownor convenient display device. The display device may take various forms,including but not limited to stereo displays suited for use in near-eyeapplications such as head mounted displays or other wearable devices.

The code stored in memory 2192 may be implemented as software and/orfirmware to program the processor(s) 2191 to carry out actions describedherein. In certain embodiments, such software or firmware may beinitially provided to the processing system 2190 by downloading from aremote system through the processing system 2190 (e.g., via networkadapter 2194).

The techniques herein may be implemented by, for example, programmablecircuitry (e.g. one or more microprocessors) programmed with softwareand/or firmware, or entirely in special-purpose hardwired(non-programmable) circuitry, or in a combination of such forms.Special-purpose hardwired circuitry may be in the form of, for example,one or more AISCs, PLDs, FPGAs, etc.

Software or firmware for use in implementing the techniques introducedhere may be stored on a machine-readable storage medium and may beexecuted by one or more general-purpose or special-purpose programmablemicroprocessors. A “machine-readable storage medium”, as the term isused herein, includes any mechanism that can store information in a formaccessible by a machine.

A machine can also be a server computer, a client computer, a personalcomputer (PC), a tablet PC, a laptop computer, a set-top box (SIB), apersonal digital assistant (PDA), a cellular telephone, an iPhone, aBlackberry, a processor, a telephone, a web appliance, a network router,switch, or bridge, or any machine capable of executing a set ofinstructions (sequential or otherwise) that specify actions to be takenby that machine.

A machine-accessible storage medium or a storage device(s) 2195includes, for example, recordable/non-recordable media (e.g., ROM; RAM;magnetic disk storage media; optical storage media; flash memorydevices; etc.), etc., or any combination thereof. The storage mediumtypically may be non-transitory or include a non-transitory device. Inthis context, a non-transitory storage medium may include a device thatis tangible, meaning that the device has a concrete physical form,although the device may change its physical state. Thus, for example,non-transitory refers to a device remaining tangible despite this changein state.

The term “logic”, as used herein, may include, for example, programmablecircuitry programmed with specific software and/or firmware,special-purpose hardwired circuitry, or a combination thereof.

The above specification, examples, and data provide a completedescription of the manufacture and use of the composition of theinvention. Since many embodiments of the invention can be made withoutdeparting from the spirit and scope of the invention, the inventionresides in the claims hereinafter appended.

The invention claimed is:
 1. A method, comprising: defining, by aprocessing device, a first coordinate in a three-dimensional (3D) spaceto display a virtual object to a viewer; displaying, by a display, thevirtual object at the first coordinate in the 3D space; determining, bya sensor, that an end-effector interacting with the virtual object at afirst point in time is pointing to a second coordinate in the 3D spacethat is different than the first coordinate where the virtual object isdisplayed; determining, by the processing device, an offset valuebetween the first coordinate where the virtual object is located and thesecond coordinate where the end-effector is pointing to, wherein theoffset value indicates a difference in the first coordinate and thesecond coordinate; determining, by the processing device, a thirdcoordinate in the 3D space, wherein the third coordinate is the firstcoordinate of the first coordinate adjusted by the offset value so thatthe viewer perceives the virtual object as being located at the firstcoordinate in the 3D space; determining, by the sensor, that theend-effector interacting with the virtual object at a second point intime is pointing to a fourth coordinate in the 3D space that isdifferent than the third coordinate where the virtual object isdisplayed; and in response to the end-effector pointing to the fourthcoordinate in the 3D space that is not the third coordinate, iterativelyadjusting, by the processing device, the fourth coordinate for thevirtual object until the end-effector points to the first coordinate. 2.The method of claim 1, further comprises displaying the virtual objectto a first display and a second display of a stereo display pair.
 3. Themethod of claim 1, further comprises: displaying the virtual object at afirst location at a first display of a stereo display pair; anddisplaying the virtual object in a second location at a second displayof the stereo display pair.
 4. The method of claim 2, wherein the firstcoordinate is a first two-dimensional coordinate and the secondcoordinate is a second two-dimensional coordinate.
 5. The method ofclaim 1, further comprises: displaying the virtual object at a firstlocation to a first eye of the viewer but not a second eye of theviewer; and displaying the virtual object at a second location to thesecond eye of the viewer but not the first eye of the viewer.
 6. Themethod of claim 5, further comprises displaying the virtual objectsimultaneously or at different points in time to the first eye of theviewer and the second eye of the viewer.
 7. The method of claim 5,further comprises displaying the virtual object sequentially to thefirst eye of the viewer and the second eye of the viewer.
 8. The methodof claim 1, further comprises: displaying nothing to a second eye of theviewer while displaying the virtual object to the first eye of theviewer; and displaying nothing to the first eye of the viewer whiledisplaying the virtual object to the second eye of the viewer.
 9. Themethod of claim 1, further comprises: obstructing or partiallyobstructing a second eye of the viewer while displaying the virtualobject to the first eye of the viewer; and obstructing or partiallyobstructing the first eye of the viewer while displaying the virtualobject to the second eye of the viewer.
 10. The method of claim 1,further comprises sensing at least one of a posture or a gesture of theviewer.
 11. A method, comprising: defining, by a processing device, afirst coordinate in an augmented reality environment or a virtualreality environment; displaying, by a display, a first virtual object toa viewer at the first coordinate; determining, by the processing device,that an end-effector interacting with the first virtual object ispointing to a second coordinate in the augmented reality environment orthe virtual reality environment, wherein the second coordinate isdifferent than the first coordinate; determining, by the processingdevice, a difference between the first coordinate where the firstvirtual object is located and the second coordinate where theend-effector is pointing to; determining, by the processing device, athird coordinate in the augmented reality environment or the virtualreality environment, wherein the third coordinate is the firstcoordinate of the first coordinate adjusted by the determineddifference; generating a second virtual object; defining, by theprocessing device, a fourth coordinate to display the second virtualobject to the viewer in the augmented reality environment or the virtualreality environment, wherein the fourth coordinate is based on thedetermined difference; and displaying, by the display, the first virtualobject at the third coordinate and the second virtual object at thefourth coordinate.
 12. The method of claim 11, further comprisingsequentially displaying the first virtual object and the second virtualobject.
 13. The method of claim 11, further comprising: displaying thefirst virtual object by a first display while outputting nothing by asecond display; and displaying the second virtual object by the seconddisplay while outputting nothing by the first display.
 14. The method ofclaim 11, further comprising: displaying the first virtual object by afirst display while obstructing or partially obstructing a seconddisplay; and displaying the second virtual object by the second displaywhile obstructing or partially obstructing the first display.
 15. Themethod of claim 11, further comprising sensing a viewer first indicationof the first virtual object and a viewer second indication of the secondvirtual object.
 16. The method of claim 11, further comprisingdisplaying the first virtual object by a first display andsimultaneously or at different points in time displaying the secondvirtual object by a second display.
 17. An apparatus comprising: a firstthree-dimensional (3D) display operable to output a virtual object at afirst coordinate in a 3D space to a first eye of a viewer; a second 3Ddisplay operable to output the virtual object at a second coordinate inthe 3D space to a second eye of the viewer, wherein the first coordinateis different than the second coordinate; a first sensor configured tomeasure a difference between a location of an end-effector in the 3Dspace interacting with the virtual object at the first coordinaterelative to the first eye of the viewer; a second sensor configured tomeasure a difference between the location of the end-effector in the 3Dspace interacting with the virtual object at the second coordinaterelative to the second eye of the viewer; and a processing devicecoupled to the first 3D display, the second 3D display, the firstsensor, and the second sensor, wherein the processing device is operableto: determine that the end-effector is pointing to a third coordinate onthe first 3D display that is different than the first coordinate wherethe virtual object is displayed at a first point in time; determine thatthe end-effector is pointing to a fourth coordinate on the second 3Ddisplay that is different than the second coordinate where the virtualobject is displayed at the first point in time; determine a firstdifference between the first coordinate where the virtual object islocated and the third coordinate where the end-effector is pointing toon a first display; determine a second difference between the secondcoordinate where the virtual object is located and the fourth coordinatewhere the end-effector is pointing to on a second display; determine afifth coordinate in the 3D space that is the first coordinate adjustedby the first difference so that the viewer perceives the virtual objectas being located at the first coordinate on the first 3D display,wherein the first 3D display is to display the virtual object at thefifth coordinate on the first 3D display at a second point in time; anddetermine a sixth coordinate in the 3D space that is the secondcoordinate adjusted by the second difference so that the viewerperceives the virtual object as being located at the second coordinateon the second 3D display, wherein the second 3D display is to displaythe virtual object at the sixth coordinate on the second 3D display atthe second point in time.
 18. The apparatus of claim 17, wherein thefirst 3D display and the second 3D display are a stereo display.
 19. Theapparatus of claim 17, wherein the first 3D display or the second 3Ddisplay comprises a see-through display.
 20. The apparatus of claim 17,wherein the first sensor or the second sensor comprises an imager.